cdm-radial: sigmet fixes #1295

michaeldiener · 2024-01-27T10:17:35Z

Support for all data types instead of just the basic 5 (power, reflectivity, width, differential reflectivity)
Universal handling for data types instead of each one separately
Most new data types are returned as raw values meaning no normalization/decoding done like for the basic 5 - see chapter 4.4
This includes support for data types that span more than one byte per bin
Added comments and small fixes

Ultimately all these changes add support for IDEAM Colombia files

Addresses the github issue #1292

I worked with this version of the Sigmet/IRIS specification: IRIS-Programming-Guide-M211318EN.pdf
(not attaching it because of copyright)

PR Checklist

Link to any issues that the PR addresses
Add labels
Open as a draft PR
until ready for review
Make sure GitHub tests pass
Mark PR as "Ready for Review"

Support for all data types instead of just 5 Universal handling for data types instead of each one separately Support for data types spanning two bytes instead of one is experimental at this point as I didn't have such a file to test Most new data types are returned as raw values meaning no normalization like done for total power, reflectivity, velocity, width, differential reflectivity Added comments and small fixes Ultimately all these changes add support for IDEAM Colombia files Addresses the github issue #1292

Code Style... :/

Another fix necessary to support IDEAM Colombia files In this case not all sweeps of num_sweeps have data, so adapting for this and handling it gracefully. Happens for the radars San_Andres and Corozal.

haileyajohnson · 2024-01-30T17:21:27Z

Hi @michaeldiener , thanks for contributing! I've marked this PR as a draft, please set it to ready for review when it's ready!

michaeldiener · 2024-01-30T18:01:47Z

Sorry for another commit today, it was a last minute fix. All done now, thank you.

michaeldiener · 2024-01-31T10:22:45Z

I'm really sorry, I have encountered another issue. Keeping this as a draft now for a few days to avoid wasting anybody's time.

Data types with more than 1 byte fully supported A few small fixes

michaeldiener · 2024-02-02T15:18:55Z

@haileyajohnson The test here seems to fail due to a connection issue when I'm not mistaken. This seems to happen in a module totally unrelated to the one I modified. Is this a known issue? How to proceed?

`dap4.test.TestHyrax > test[5: nc4_strings_comp.nc] STANDARD_ERROR
Comparison: vs /github/workspace/dap4/src/test/data/resources/baselinehyrax/nc4_strings_comp.nc.ncdump
>>>> 21 CHANGED FROM
station =
{ "one", "two", "three", "four", "five", "one_b", "two_b", "three_b", "four_b", "five_b", "one_c", "two_c", "three_c", "four_c", "five_c", "one", "two", "three", "four", "five", "one", "two", "three", "four", "five", "one_f", "two_f", "three_f", "four_f", "five_f"
}
scan_line = "r", "r1", "r2", "r3", "r4"
codec_name = "mp3"
lat =
{0, 10, 20, 30, 40, 50}
lon =
{-140, -118, -96, -84, -52}
}
>>>> CHANGED TO
dap4.core.util.DapException: Request failure: 500: http://test.opendap.org/opendap/nc4_test_files/nc4_strings_comp.nc.dap?dap4.checksum=true
>>>> Dap4 Testing: End of differences.

dap4.test.TestHyrax > test[5: nc4_strings_comp.nc] FAILED
java.lang.AssertionError: *** FAIL
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.assertTrue(Assert.java:42)
at dap4.test.TestHyrax.test(TestHyrax.java:186)`

haileyajohnson · 2024-02-02T15:53:53Z

@michaeldiener that looks like a flaky test. I'll rerun your tests and if it fails again I'll turn that one off.

haileyajohnson · 2024-02-02T18:00:40Z

we're taking a look at the flaky tests but feel free to mark this ready when you want us to review it

michaeldiener · 2024-02-02T19:26:50Z

@haileyajohnson Thank you! I'm going to wait for a few more days, just to make sure everything keeps running smoothly in my own server environment.
Also I'm trying to find more Sigmet IRIS example files, especially ones that contain data types with more than one byte per bin, e.g. DB_DBZ2 Reflectivity (2 byte). This seems harder than anticipated. I sent a support request to Vaisala, they are the ones that create the original Sigmet IRIS software, but not sure they will respond as I'm not one of their customers. Other than that I'm not sure where I can find a file like this.

Fix for 2 byte data types (was able to find an example file finally)

…_sigmet

Fix for reading data from ray for only the number of bins available

michaeldiener · 2024-02-05T09:16:05Z

Found an example of a file I was looking for eventually. All working well now IMHO.

Support for all data types instead of just 5 Universal handling for data types instead of each one separately Support for data types spanning two bytes instead of one is experimental at this point as I didn't have such a file to test Most new data types are returned as raw values meaning no normalization like done for total power, reflectivity, velocity, width, differential reflectivity Added comments and small fixes Ultimately all these changes add support for IDEAM Colombia files Addresses the github issue #1292

Code Style... :/

Another fix necessary to support IDEAM Colombia files In this case not all sweeps of num_sweeps have data, so adapting for this and handling it gracefully. Happens for the radars San_Andres and Corozal.

Data types with more than 1 byte fully supported A few small fixes

Fix for 2 byte data types (was able to find an example file finally)

Fix for reading data from ray for only the number of bins available

…_sigmet

haileyajohnson · 2024-03-11T21:16:00Z

@michaeldiener I pushed a few changes to your branch and I'm gonna run it against our full test suite, but if that passes I'd say it's ready to go. Thanks again for contributing!

michaeldiener · 2024-03-12T08:53:52Z

Please let me go through your changes before merging. At first sight there are a few things I want to have a closer look.

Fixes for change "PR feedback and cleanup" b89cf77

michaeldiener · 2024-03-12T09:29:40Z

@haileyajohnson Thanks for the review!
Unfortunately some of your optimizations introduced 2 new bugs and I have resolved this in my latest commit.

haileyajohnson · 2024-03-12T15:35:18Z

Thanks! We're having some technical difficulties with our test data at the moment, so I'm waiting on that to resolve to run our tests :)

michaeldiener requested a review from haileyajohnson as a code owner January 27, 2024 10:17

michaeldiener added 2 commits January 27, 2024 11:30

cdm-radial: sigmet

ec7cab2

Code Style... :/

cdm-radial: sigmet

0c07e0b

Another fix necessary to support IDEAM Colombia files In this case not all sweeps of num_sweeps have data, so adapting for this and handling it gracefully. Happens for the radars San_Andres and Corozal.

haileyajohnson marked this pull request as draft January 30, 2024 17:20

michaeldiener marked this pull request as ready for review January 30, 2024 18:01

michaeldiener marked this pull request as draft January 31, 2024 10:21

michaeldiener added 2 commits February 2, 2024 15:19

cdm-radial: sigmet

e3b35f7

Data types with more than 1 byte fully supported A few small fixes

Merge branch 'maint-5.x' into maint-5.x_sigmet

e6bef22

michaeldiener added 5 commits February 3, 2024 11:08

cdm-radial: sigmet

e3a19e1

Fix for 2 byte data types (was able to find an example file finally)

Merge remote-tracking branch 'origin/maint-5.x_sigmet' into maint-5.x…

a96caac

…_sigmet

Merge branch 'maint-5.x' into maint-5.x_sigmet

2e17ad6

code style... :/

2d89216

cdm-radial: sigmet

d597a4e

Fix for reading data from ray for only the number of bins available

michaeldiener marked this pull request as ready for review February 5, 2024 09:16

michaeldiener added 7 commits February 5, 2024 08:46

cdm-radial: sigmet

97ca4ed

Code Style... :/

cdm-radial: sigmet

c5b66c6

Another fix necessary to support IDEAM Colombia files In this case not all sweeps of num_sweeps have data, so adapting for this and handling it gracefully. Happens for the radars San_Andres and Corozal.

cdm-radial: sigmet

6562aa6

Data types with more than 1 byte fully supported A few small fixes

cdm-radial: sigmet

9e09321

Fix for 2 byte data types (was able to find an example file finally)

code style... :/

1d671cb

cdm-radial: sigmet

b70e345

Fix for reading data from ray for only the number of bins available

michaeldiener and others added 3 commits March 11, 2024 09:02

Merge remote-tracking branch 'origin/maint-5.x_sigmet' into maint-5.x…

14e430b

…_sigmet

Merge branch 'maint-5.x' into maint-5.x_sigmet

343faf1

PR feedback and cleanup

b89cf77

cdm-radial: sigmet

d17603c

Fixes for change "PR feedback and cleanup" b89cf77

Merge branch 'maint-5.x' into maint-5.x_sigmet

3707ddd

haileyajohnson approved these changes Mar 14, 2024

View reviewed changes

haileyajohnson merged commit 3fb7e4c into Unidata:maint-5.x Mar 14, 2024
10 checks passed

michaeldiener deleted the maint-5.x_sigmet branch March 14, 2024 07:18

michaeldiener mentioned this pull request Mar 18, 2024

cdm-radial: Parsing issues with radar files from IDEAM Colombia #1292

Closed

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cdm-radial: sigmet fixes #1295

cdm-radial: sigmet fixes #1295

michaeldiener commented Jan 27, 2024 •

edited

Loading

haileyajohnson commented Jan 30, 2024

michaeldiener commented Jan 30, 2024

michaeldiener commented Jan 31, 2024

michaeldiener commented Feb 2, 2024

haileyajohnson commented Feb 2, 2024

haileyajohnson commented Feb 2, 2024

michaeldiener commented Feb 2, 2024

michaeldiener commented Feb 5, 2024

haileyajohnson commented Mar 11, 2024

michaeldiener commented Mar 12, 2024

michaeldiener commented Mar 12, 2024

haileyajohnson commented Mar 12, 2024

cdm-radial: sigmet fixes #1295

cdm-radial: sigmet fixes #1295

Conversation

michaeldiener commented Jan 27, 2024 • edited Loading

PR Checklist

haileyajohnson commented Jan 30, 2024

michaeldiener commented Jan 30, 2024

michaeldiener commented Jan 31, 2024

michaeldiener commented Feb 2, 2024

haileyajohnson commented Feb 2, 2024

haileyajohnson commented Feb 2, 2024

michaeldiener commented Feb 2, 2024

michaeldiener commented Feb 5, 2024

haileyajohnson commented Mar 11, 2024

michaeldiener commented Mar 12, 2024

michaeldiener commented Mar 12, 2024

haileyajohnson commented Mar 12, 2024

michaeldiener commented Jan 27, 2024 •

edited

Loading